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Summary 


From  research  conducted  with  23  trainee  sections  at  the  Naval  OCS 
in  Newport,  data  are  presented  relative  to  the  reliability  of  peer 
nominations  as  it  is  affected  by  three  variables:  the  period  of  time 
the  group  has  spent  together;  the  nature  of  the  set  given,  i.e.,  "for 
research  purposes"  or  "for  administrative  purposes";  and,  the  quality  or 
characteristic  to  Le  evaluated  by  the  nominator  in  making  his  nominations. 
The  following  major  findings  are  reported: 

1.  The  reliability  of  forms  administered  very  early  in 
training — after  the  groups  have  been  together  for  four  to  five 
days — is  a  reasonable  approximation  of  the  reliability 
obtained  with  the  same  forma  and  the  same  groups  at  later 
points  in  training. 

2.  The  peer  nomination  scores  obtained  at  the  end  of 
the  third  week  of  training  correlate  at  a  high  level  with 
those  scores  obtained  on  the  same  groups  at  a  later  level, 
i.e.,  sixth  week. 

3.  There  is  no  significant  difference  in  the  single- 
stage  reliability  or  longitudinal  reliability  of  comparable 
forms  administered  under  the  "research"  as  against  the  "admin¬ 
istrative"  set. 

A.  all  forms — irrespective  of  the  four  different  rated 
qualities  involved — show  a  tendency  to  begin  with,  substantial 
reliability  and  rise  in  subsequent  administrations  to  only  a 
slightly  higher  plateau. 

While  practical  implications  will  be  handled  more  comprehensively 
in  later  reports,  it  is  concluded  here  that  a  peer  nomination  administered 
as  early  as  the  third  week  of  training  will  yield  substantially  the  same 
information  as  th it  which  is  now  obtained  at  the  sixth  week,  or  even  later. 
Furthermore,  the  data  suggest  that  the  "administrative"  set  leads  to 
neither  more  nor  less  reliable  scores  than  those  secured  through  the 
presumably  less  threatening,  or  more  lightly-taken,  "research"  set. 


I.  Introduction 


This  is  the  first  in  a  series  of  reports  to  deal  with  a  large- 
scale  study  of  peer  ratings  completed  at  the  U.  3.  Naval  School, 

Officer  Candidate  (0C3)  in  Newport,  Rhode  Island,  during  the  summer 
and  fall  of  1955. 

At  the  very  outset,  it  is  of  significance  to  note  that  this  study 
represented  an  expansion,  rather  than  a  replication,  of  the  previous 
work  in  this  sphere  done  at  Newport  by  the  American  Institute  for 
Research  under  Navy  Contract  Nonr  890(01).  While  the  area  of  concern 
of  the  current  project  was  somewhat  more  circumscribed  than  that  of  the 
previous  project,  we  are  appreciative  of  the  basic  data  available  to 
us  from  the  work  this  group  had  already  completed. 

In  simplest  practical  terms,  the  project  was  conceived  to  eventually 
provide  more  information  regarding  the  application  of  peer  ratings— 
specifically,  "peer  nominations"— as  a  supplemental  screening  device 
in  the  OCS.  Although  this  issue  will  be  touched  upon  here,  the  final 
project  report  will  serve  as  the  vehicle  for  a  more  Comprehensive 
delineation  of  the  relevant  implications  along  this  line. 

The  research  objectives  of  this  study,  basically,  were  to  determine 
the  reliability  and  validity  of  peer  ratings  under  various  instructional 
sets  and  time  exposures.  Certain  of  the  answers  regarding  ultimate 
validity  must  necessarily  await  the  establishment  of  post-training 
criteria,  at  this  time,  however,  it  is  both  appropriate  and  feasible 
to  report  on  the  data  available  with  regard  to  the  reliability  of  the 
peer  rating  forms  used. 


II.  Background  of  the  Problem 

In  practice,  the  so-called  peer  rating  involves  each  group  member's 
evaluation  of  his  peers  on  some  recognizable  quality  or  set  of  charac¬ 
teristics  which  are  manifested  directly,  or  inferrable  indirectly,  from 
day-to-day,  personal  interactions.  Such  individual  evaluations  are 
then  integrated  into  a  composite  score  reflecting  each  person's  standing 
in  his  group.  The  advantage  of  this  technique  appears  to  rest  in  its 
ability  to  yield  unique  supplementary  data  drawn  from  intimate  contact 
among  the  personnel  involved. 

Peer  rating  procedures  have  already  demonstrated  a  relatively 
high  order  of  predictability  against  various  performance  criteria,  with 
a  substantial  numoer  of  military  groups  (e.g.,  4,  6,  7,  B,  11,  12,  13, 

14,  15).  It  may  be  noted,  too,  that  the  greatest  development  of  the 
peer  ratings  has  taken  place  within  the  military  services  (3).  Several 
questions  remain  to  be  answered,  however,  before  optimum  utilization 
of  peer  ratings  may  be  effected. 

III.  Statement  of  the  Problem 

For  purposes  of  this  interim  report,  three  core  problems  may  be 
specified  for  consideration.  These  involve,  first,  the  length  of  time 
which  the  group  must  have  spent  together  before  peer  ratings  will 
approximate  maximized  validity  and  reliability;  second,  the  presence  of 
any  differential  effects  on  validity  and  reliability  accruing  from  the 
use  of  peer  ratings  with  a  "research"  set  as  against  those  with  a  "real" 
set;  and,  finally,  the  variations  in  validity  and  reliability  which  may 
be  attributable  to  the  nature  of  the  quality  on  which  the  rtter  is 
instructed  to  rate  the  ratee. 


3 


as  has  been  suggested,  this  report  will  confine  itself  to  data 
bearing  on  the  reliability  aspect  of  these  questions.  The  matter  of 
validity  will  be  appropriately  considered  in  a  subsequent  report. 

IV.  Subjects  and  Setting 

Through  the  joint  cooperation  of  the  Office  of  Naval  Research,  the 
Bureau  of  Naval  Personnel,  and  the  Naval  OCS  itself,  the  entire  input 
of  OCo  Class  23  was  made  available  for  this  study.  The  sample  entered 
training  in  July  of  1955  and  consisted  of  23  sections  numbering  about 
jO  men  each;  there  is  no  reason  to  suppose  that  assignment  to  these 
sections  was  on  anything  other  than  a  random  basis.  The  total  N  avail¬ 
able  for  study  exceeded  700,  at  the  beginning  of  this  project. 

The  program  at  the  0C3  is  of  sixteen  weeks  duration,  with  an 
orientation  week  introduced  before  the  actual  onset  of  the  training 
cycle.  During  this  one-week  period,  student  personnel  are  assigned  to 
sections,  receive  books  and  clothing,  take  classification  tests,  receive 
orientation  lectures,  but  do  not  attend  formal  classes,  as  such. 

Except  for  a  small  minority  drawn  from  the  fleet— in  this  class 
numbering  fewer  than  5/®— all  of  the  students  are  graduates  of  four 
year  college  programs.  The  mean  age  of  this  class  was  22  years  with 
only  a  minimal  dispersion  above  this  fig are. 

Students  at  the  OCJ  are  selected  according  to  rigorous  mental  and 
physical  standards,  all  are  volunteers,  and  must  agree  to  remain  on 
active  duty  as  officers  for  three  years  following  the  successful  con¬ 
clusion  of  training. 

V.  Instruments 

The  previous  work  of  Suci,  Vallance,  and  Glickmar.  (9)  established 
several  points  which  bore  upon  the  selection  of  instruments  for  this  study 
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Their  research  indicates  that  "...  ratings  by  peers  based  on  either 
the  behavior  at  OCS  or  projected,  future  behavior  are  equally  reli¬ 
able  ...»  (9,  p.ll) ;  in  addition,  they  report  that,  "The  technique 
which  requires  selection  of  the  upper  and  lower  segments  appears  to 
have  as  satisfactory  reliability  as  any  of  the  other  tested  tech¬ 
niques  ..."  (9,  p.ll);  finally,  they  note  that  the  order  of  presen¬ 
tation  which  yields  the  lowest  correlation  between  forms  is  "future 
officer"  followed  by  "OCJ  behavior"  (9,  p.12). 

Four  basic  sociometric  forms  of  the  peer  nomination  variety  were 
utilized.  Based  upon  the  research  cited  above,  a  primary  form  was 
utilized  with  all  sections  which  called  for  nominations  on  "success 
as  a  future  Naval  Officer"  (FO) .  This  form  was  seen  to  be  of  par¬ 
ticular  worth  in  its  likely  prediction  of  more  distant,  fleet  perform¬ 
ance  criteria.  In  addition  to  this  primary  form,  each  section  received 
one  of  three  so-called  secondary  forms,  i.e.,  "leadership  qualities"  (LQ), 
"interest  in  and  enthusiasm  for  the  Naval  Service"  (IK),  and  "proba¬ 
bility  of  success  in  OCS"  (OC).  The  selection  of  these  forms  rested 
upon  a  need  to  tap  those  characteristics  which  might  relate  to  both 
in-training  and  post-training  performance — i.e.,  interpersonal  qualities, 
motivation,  and  ability  having  evident  relevance  to  OCS  performance. 

Cutting  across  this  pattern,  approximately  half  the  sections 
received  a  "research"  set  with  the  ex;, licit  point,  appearing  on  their 
peer  nomination  forms,  that  "The  results  of  these  ratings  are  to  be 
used  for  research  purposes  only  and  will  not  affect  your  Navy  career." 

The  other  sections  were  given  explicit  instructions  that  "The  results 
of  these  ratings  may  be  used  for  administrative  purposes."  This  split 
in  treatment  was  designed  to  provide  data  on  the  differential  reliability 
ini  validity  resulting  from  administration  under  a  "research"  set  as 
against  an  "administrative"  set. 
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In  all  there  were  eight  possible  forms,  i.e.,  four  characteristics 
to  be  rated  times  two  sets.  In  appendix  A,  form  FO-HO  is  reproduced 
as  an  illustration  of  the  format  followed  with  all  forms.  Since  this 
was  varied  only  slightly  to  accommodate  alternative  instructions,  the 
reader  may  view  this  as  an  example  of  the  general  instructions  applied. 

As  will  be  noted,  all  of  the  forms  required  five  "high"  and  five 
"low"  nominations  in  order  of  preference.  Each  of  the  subjects  was 
provided  with  a  complete  alphabetical  roster  of  his  sectionmates  every 
time  he  was  required  to  complete  a  form.  The  principal  investigator  was 
the  sole  administrator  of  the  forms  for  all  sections  at  all  times. 

VI.  Study  Design 

The  ultimate  design  of  the  study  is  reflected  in  Table  1.  it 
will  be  seen  that  the  23  sections  were  divided  into  six  blocks  of  four 
sections  each,  except  for  one  block  which,  of  necessity,  was  limited 
to  three  sections.  Sections  were  assigned  to  blocks  on  a  rotation 
basis  from  the  five  companies  in  the  second  battalion  which  comprised 
the  class,  i.e..  How,  Jig,  King,  Love,  Mike.  Such  differences  as 
might  exist  Letween  companies  were  thus  restricted  in  their  conceivable 
ability  to  contajninate  the  study  design. 

Once  having  been  assigned  to  a  given  block,  the  treatment  of  any 
given  section  was  identical  through  training.  Three  major  adminis¬ 
trations  of  these  forms  was  carried  on  during  the  training  cycle:  the 
first  occurred  during  the  so-called  "orientation  week" — refer-ed  to 
hereafter  as  the  "0"  week — after  the  subjects  had  been  together  in 
their  respective  sections  for  fo  ir  to  five  days;  at  the  end  of  the  third 
week  of  training  ("3"  week);  and,  at  the  end  of  the  sixth  week  of 
training  ("6"  week).  The  design  appearing  in  Table  1  was  replicated. 


TABLE  I 


I 

I 

l 

I 

J 

1 


Forms 


Study  Design*  Indicating 
Used  and  Sections  to  Which  Applied 


Section  Allocation 


- - 

- w  w  I 

j - - - - Set  j 

Forms** 

Research  (RO) 

Administrative  (AU) 

"Future  Officer"  (FO) 
"Interest  Ac  Enthusiasm"  (IE) 

R~lj  U-l,  L-l,  M— 1 

A/ 

H-2,  K-l,  L-2,  M-2 

B/ 

"Future  Officer"  (FO) 

"Success  in  0C3»  (OC) 

H-3,  H-4,  J-4^  k-4 

c/ 

H-5,  J-5,  K-5 

IV 

"Future  Officer"  (FO) 
"Leadership  Qualities"  (LQ) 

J-2,  h-2,  L-3,  M-3 

E/ 

K- 3,  L-4,  M— 4 

F/ 

IMS  ££*,£££ 

omission  of  the  secondary  foms-the  thirteenth  week. 


**Note  that  each  section  received  two  forms. 
/Signifies  group  code  designation. 
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therefore,  a  total  of  three  times,  at  the  end  of  the  thirteenth  week 
of  training,  another  administration  of  forms  was  made,  but  this  last 
time  only  the  primary,  "future  officer,"  form  was  used.  In  all  other 
respects,  the  design  was  identical  for  the  latter  administration. 

VII.  Scoring 


Following  the  pattern  utilized  in  several  studies  elsewhere  (e.g.,  4, 
a  direct  weighting  procedure  was  applied  to  derive  peer  nomination  scores. 
The  highest  nominee  was  awarded  a  plus  5,  the  next  highest,  a  plus  4, 
and  so  on  through  the  five  "highs";  similarly,  the  lowest  nominee  was 
assigned  a  minus  5,  the  next  lowest,  a  minus  4,  and  so  on.  ,.n  algebraic 
sum  was  then  obtained  for  each  subject  and  divided  by  the  N  of  the  group 
minus  1,  since  no  subject  may  nominate  himself.  This  results  in  an 
average  score  ranging  on  a  continuum  from  paus  five  to  minus  five.  To 
remove  the  minus  sign,  a  constant  of  5  *.ad  added  to  this  score;  the 
resultant  value  was  then  multiplied  by  10  in  order  to  permit  the  use  of 
a  two  digit  score  without  the  intervening  decimal  point.  'Where  a  subject 
had  an  average  raw  value  of  minus  5,  and  hence  a  score  of  00,  he  was 
arbitrarily  given  a  score  of  01,  after  the  constant  had  been  added  and 
the  multiplication  had  taken  place,  at  the  other  end  of  the  range,  the 
plus  5  subject  was  given  a  score  of  99,  rather  than  100. 

The  distribution  arising  from  this  procedure  has  normal  charac¬ 
teristics  with  a  range  from  1  to  99,  a  nuan  of  50,  an-  a  standard 
deviation  approximating  10,  for  the  total  population  of  the  study. 

While  this  score  may  be  seen  to  have  certain  features  of  a  standard 
score,  it  does  not  tend  to  obscure  section  differences  so  much  as  does 
the  st mdard  score.  Furthermore,  this  score  does  not  presume  the 
homogeneity  of  characteristics  from  section  to  section  which  is  assumed 
under  a  generalized  standard  score  transformation. 
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VIII.  Determination  of  Reliability 

Two  major  approaches  may  be  followed  in  determining  the  reliability 
of  peer  nomination  data.  In  the  first  place,  one  may  focus  attention  on 
the  internal  consistency  of  the  nominations  made  within  a  given  group  at 
some  discrete  point  in  time.  In  the  second  place,  one  may  deal  with  their 
consistency  over  ti  te,  as  reflected  in  repeat  administrations. 

Of  the  two  approaches,  the  former  is  more  typically  up. lied.  The 
latter  approach  has  evident  disadvantages  in  that  time  exposure  is  very 
likely  to  have  an  impact  in  altering  the  position  of  subjects  on  the 
status  continuum.  This  raises  the  question  of  whether,  indeed,  a  "good" 
peer  nomination  form  ought  to  have  high  repeat  reliability;  in  point  of 
fact,  one  might  wish  to  use  peer  nominations  precisely  for  a  study  of 
temporal  fluctuations  in  status,  a.  well  as  the  extent  to  which  status  is 
maintained.  Thus,  a  low-level  correlation  between  scores  yielded  by  two 
administrations  of  the  same  peer  nomination  form  is  very  often  evidence 
of  an  unstable  group  p  ttern  rather  than  of  an  inherent  unreliability 
attributable  to  the  form  itself. 

For  our  own  purposes,  we  find  both  varieties  of  reliability  of  concern 
since  we  shall  wish  to  know  the  internal  consistency  of  scores  obtained 
from  various  forms  at  various  times,  and  the  rela  ionship  over  time  of 
scores  obtained  from  two  administrations  of  the  same  form,  uoth  consid¬ 
erations  bear  upon  osr  problem. 

In  practice,  the  calculation  of  single-stage  reliability,  or  internal 
consistency,  of  a  peer  nomination  score  is  normally  unUertaken  by  an  odd- 
even  split  of  the  raters  within  the  roup  so  as  to  afford  two  meas  .res  of 
status.  The  correlation  between  these  measures  io  then  treated  by  the 
Spearman-Brown  formula  to  yield  a  corrected  reliauility  coefficient. 
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IX.  Results  and  Discussion 

Table  II  reports  the  corrected  split-half  reliability  coefficients 
calculated  at  various  points  in  the  life  cycle  of  relevant  groups  for  the 
eight  peer  nomination  forms  utilized.  The  reduced  N's  are  due  to  the  fact 
that  these  coefficients  were  based  upon  a  random  selection  of  representative 
sections  receiving  the  treatments  involved.  The  identical  sections  are 
treated  at  each  time  period  so  as  to  control  section  variations  which 
might  obscure  time  effects. 

It  will  be  seen  that  the  reliabilities  in  the  first  column,  for  all 
forms,  approximate  .90— even  though  the  sections  had  been  comprised  only 
four  or  five  days  before.  Omitting  considerations  of  validity,  it  is 
striking  to  note  the  rapidity  with  which  a  *roup  perception  of  individuals 
appears  to  have  crystallized.  This  high  reliability  is  of  particular 
note  when  one  considers  that  previous  studies,  based  upon  peer  nomination 
scores  drawn  from  later  weeks  of  training,  show  r's  which  are  not  signifi¬ 
cantly  greater  (3).  This  is  also  reflected  in  our  data  here. 

The  yield,  as  regards  higher  reliabilities,  is  greater,  but  not 
significantly  so,  as  one  proceeds  to  later  time  periods.  It  would  appear 
that  the  major  increase  occurs  from  the  orientati >n  week  to  the  third 
week,  after  which  the  coefficients  are  stabilized.  This  is  particularly 
discernible  in  the  case  of  the  F0  forms  which  were  carried  through  to  a 
thirteenth  week  administration. 

With  respect  to  the  reliability  of  comparable  forms  administered 
under  a  "research"  (ftO)  as  opposed  to  an  "administrative"  (  vb)  set,  no 
differences  of  a  significant  magnitude  may  be  noted  at  any  time  level. 

Their  respective  patterns  are  practically  identical. 
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TABLE  II 

Corrected  Split-Half  Reliabilities  of 
Eight  Peer  Nomination  Forms  at 
Various  Stages  of  Training 


Week  in  Training 


Forms 

Orientation 

Third 

Sixth 

j  Thirteenth 

Week 

Week 

Week 

Week 

FO-RO 

.94 

.97 

.97 

•  97 

96* 

92 

91 

91 

FO-AU 

•  94 

.97 

.97 

.95 

96 

96 

95 

91 

IE-RO 

.91 

.96 

.96 

32 

29 

29 

I£-AU 

.88 

.89 

.94 

32 

32 

32 

LQ-RO 

.93 

.96 

.96 

32 

32 

31 

LQ-AU 

.94 

.97 

.98 

32 

32 

32 

OC-RO 

.93 

.98 

-99 

32 

31 

31 

OC-AU 

.89 

.97 

.98 

32 

32 

31 

*The  number  below  each  coefficient  indicates 
the  N  upon  which  it  is  based. 
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A  contrast  between  the  intercorrelation  matrix  for  the  "future 
officer"  form  administered  at  various  stages  under  an  RO  set  and  the 
matrix  for  the  same  form  under  an  AU  set  is  to  be  seen  in  Table  III. 

In  most  respects  the  nr  trices  are  quite  comparable.  Both  indicate  a 
sequential  decrease  in  correlation  between  the  orientation  week  scores 
and  those  scores  obtained  from  later  administrations.  Both  ar ;  notable 
for  the  high  correlation,  i.e.,  .90  and  .94  respectively,  between  scores 
derived  from  the  third  and  sixth  week  administrations.  in  general,  the 
pattern  for  both  matrices  reflects  a  stability  of  measure  from  the  third 
week  administration  onward.  Except  for  a  significance  of  difference 
beyond  the  5.*  level— following  a  z  transf ormation  for  the  two  coefficients 
just  noted— none  of  the  other  pairings  are  significantly  different.  Thus, 
there  is  no  broad  disparity  between  the  two  matrices  indicative  of  a 
differential  effect  on  reliability  attributable  to  the  two  sets  under  study. 

Tables  IV,  V,  and  VI  follow  on  the  pattern  established  in  Table  III. 

In  each  case,  two  matrices  are  provided — one  for  the  RO  set  and  one  for 
the  AU  set— indicating  the  intercorrelation  of  comparable  secondary  forms 
administered  at  three  time  levels;  in  each  instance,  these  have  been 
confined  within  a  triangle.  Data  are  also  presented  indie  ting  the  re¬ 
lationship  of  each  secondary  form  to  its  prisuiry  (F0)  counterpart  at 
each  time  level. 

Taken  generally,  the  differences  between  comparable  coefficients 
in  the  upper  (RO)  and  lower  (AU)  matrices  of  these  tables  are  not  statis¬ 
tically  significant  for  the  three  secondary  forms.  There  is  one  exception 
to  this,  however;  the  orientation  week  correlation  of  F0-R0  and  L^-RO  is 
significantly  higher  than  the  parallel  correlation  of  F0-au  and  Ly-.iU 
(Tatle  VI). 
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TABLE  III 


+h«  MP  f  rT®^ftlon  of  Peer  Gemination  Scores  for 

Future  Officer"  Form  Administered  Under  Two  Sets 
Independently  at  Four  Stages  of  Training 


N  throughout  equals  349 


FO-AU  "3"  Week  FO-AU  "6"  'Week 

FO-AU  "0"  Week 
FO-AU  "3"  Week 
FO-AU  "6"  Week 
FO-AU  "13"  Week 


FO-AU  "13"  Week 


.  .56 


.S3 


•  91 


N  throughout  equals  320 


fVeril?e  r'l  were  emulated  from  the  triad 
by  application  of  Fisher's  z  transformation. 
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TABLE  IV 


mtercorrelation  of  Peer  Nomination  Scores  for 
the  Interest  and  Enthusiasm"  Form 
Administered  Under  Two  Sets  Independently  at 
Three  Stages  of  Training  With 
Relationship  to  "Future  Officer"  Score*  ^ 


FO-RO  "0"  Week 
IE-RO  "0"  Week 
FO-RO  "3"  'Week 
IE-RO  "3"  Week 
FO-RO  "6"  Week 
IE-RO  "6"  Week 


.65 


IE-AU  "0"  Week 
FO-AU  "0"  Week  .86 

IE-AU  "0"  Week  _ 

FO-AU  "3"  Week 
IE-AU  "3"  Week 
FO-AU  "6"  Week 
IE-AU  "6"  Week 


IE-AU  "3"  Week  IE-aU  "6"  Week 


•  75 


*These  average  r's  were  calculated  from  the 
by  application  of  Fisher's  z  transformation 


triad 
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TABLE  V 


Intercorrelation  of  Peer  Nomination  Scores  for 
the  "Success  in  OCJ"  Form 
Administered  Linder  Two  Sets  Independently  at 
Three  Stages  of  Training  With 
Relationship  to  "Future  Officer"  Scores  at  Each  Stage 


OC-RO  "0"  Week 
F0-R0  "0"  Week  .68 

OC-RO  "0"  Week 

FO-RO  "3"  Week 

OC-RO  "3"  Week 

FO-RO  "6"  Week 

OC-RO  "6"  Week 


OC-RO  "3"  Week  OC-RO  "6"  Week 


N  throughout  equals  112 


FO-AU  "0"  Week 
OC-AU  "0"  Week 
FO-AU  "3"  'Week 
OC-aU  "3"  Week 
FO-AU  "6"  Wee k 
OC-aU  "6"  Week 


OC-AU  "0"  Week  OC-aU  "3"  Wee’:  OC-AU  "6"  Week 

.82 


N  throughout  equals  82 


*These  average  r's  were  calculated  from  the  triad 
by  application  of  Fisher's  z  transformation. 
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TaBLE  VI 


Intercorrelation  of  Peer  Nomination  Scores  for 
the  "Leadership  Qualities"  Form 
Administered  Under  Two  Sets  Independently  at 
Three  Stages  of  Training  With 
Relationship  to  "Future  Officer"  Scores  at  Each  Stage 


LQ-RO  "0"  Week 

LQ-RO  "3"  Week  LQ-RO  "6" 

Week 

FO-RO  "0"  Week 

.91 

LQ-RO  "0"  Week 

— 

^^<82  .77 

FO-RO  "3"  Week 

LQ-RO  "3"  Week 

FO-RO  "6"  Week 

.96 

LQ-RO  "6"  Week 

_ 

N  throughout  equals  118 

IQ-aU  "0"  Week 

LQ-aU  "3"  Week  LQ-AU  "6" 

Week 

FO-Aj  "0"  Week 

.75 

LQ-aU  "0"  Week 

— 

"*^>^85  .78 

FO-AU  "3"  Week 

LQ-aU  "3"  Week 

1 

| 

/ 

VJl 

FO-AU  "6"  Week 

^4 

.97 

LQ-aU  "6"  Week 

— - 

N  throughout  equals  122 

*These  average  r's  were  calculated  from  the  triad 
by  application  of  Fisher's  z  transformation. 
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Considering  now  the  triad  of  r's  representing  the  intercorrelation 
of  each  secondary  form  with  itself  at  each  time  level,  we  find  that  the 
high  relationship  (about  .90)  between  third  and  sixth  week  nomination 
scores  is  maintained  for  all  forms,  under  both  sets.  Furthermore,  it 
is  possible  to  calculate  an  average  intercorrelation  for  each  triad  so 
as  to  compare  the  stability  of  scores  for  the  various  forms  and  sets  at 
various  times,  using  a  z  transformation  of  r'3,  this  has  been  done.  These 
average  r's  are  presented  in  the  center  of  each  triad.  A  t  test  of  these 
reveals  no  significant  difference  between  sets  for  any  of  the  forms, 
including  F0,  where  only  its  first  three  administrations  are  considered. 

Of  &11  the  secondary  forms,  LQ  shows  the  highest  average  intercorrelation 
of  the  three  scores.  This  is  significantly  greater  beyond  the  5£  level 
for  LQ-RO  against  0C-R0,  and  for  LQ-AU  against  both  OC-AU  and  I&-AU.  The 
average  intercorrelation  for  FO's  first  three  administrations  under  both 
sets  (Table  Ill),  is  not  significantly  different  from  that  found  for  any 
of  the  secondary  forms.  If,  however,  any  one  of  the  secondary  forms  is  to 
be  considered  to  yield  the  most  consistent  score  over  time,  then  this  is 
the  LQ  form.  Since  the  evidence  is  somewhat  spotty,  and  since  consistency 
may  have  its  limitations,  it  would  not  do  to  suggest  that  the  LQ  is  per¬ 
force  the  "best"  form. 

In  Table  VI  it  will  be  seen  that  the  LQ  scores  have  a  substantially 
higher  intercorrelation  with  F0  scores  from  each  time  period  than  do  the 
other  two  secondary  scores  (Cf.  Tables  IV  and  V);  this  difference  for  LQ 
against  both  IE  and  0C  is  significant  beyond  the  level  for  all  three 
administrations  of  LQ  under  the  RQ  set,  and  for  the  latter  two  acLidnistra- 
tions  under  an  AU  set.  From  the  magnitude  of  the  r's  involved,  it  would 
appear  that  the  F0  and  LQ  forms  are  being  perceived  as  essentially 


similar. 


17 


X.  Conclusions 

The  data  available  from  both  varieties  of  reliability  determination 
are  essentially  mutually  supportive.  The  internal  consistency  of  measure 
is  high  throughout  the  time  sequence.  Early  nominations  manifest  a 

significant  relationship  to  later  nominations,  by  the  same  groups,  with 
the  same  forms. 


ny  the  third  week— and  perhaps 


,  iiau  we  LaKen 


the  nomination  score  is  stabilized,  at  least  insofar  as  its  correlation 
with  the  sixth  week  score  is  concerned.  In  the.  case  of  our  primary  form  (FO), 
where  an  additional  administration  is  available,  the  third  week  score  is 
significantly  lower  in  relationship  to  the  thirteenth  week  score  than  it 

is  to  the  sixth  week  score.  This  does  not  necessarily  vitiate  the  validity 
of  the  early  reading,  however. 


Of  particular  interest  is  the  question  of  the  eventual  worth  of  the 
very  early,  i.e.,  orientation  week,  ratings  obtained  here.  While  their 
relationship  to  later  ratings  is  high,  there  is  certainly  not  a  one-to-one 
correspondence  evident.  That  this  should  be  so  is  understandable  in  view 
of  the  greater  range  of  information  available  to  the  rater  at  later 
stages.  Indeed,  we  may  speculate  that  the  essential  virtue  of  the  early 
rating  is  that  it  is  based  upon  personal  contact  without  the  direct 
intrusion  of  academic  performance  considerations,  which  usually  correlate 
appreciably  with  peer  nominations. 


In  this  regard,  it  is  of  interest  to  note  the  high  temporal  stability 
of  the  leadership  ratin ;,  which  tends  to  be  founded  in  an  "interpersonal 
quality"  rather  than  an  academically-loaded  performance  characteristic. 
Again,  what  this  means  in  validity  terms  is  an  open  question  for  the  time. 
It  is  probable,  however,  that  this  has  implications  regarding  the  con¬ 
tribution  of  unique  variance  in  the  prediction  of  more  ulti  lte  criteria. 
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Added  to  this  picture,  too,  is  the  relatively  low  level  average  inter- 
correlation  of  the  OC  scores,  which  are  very  likely  subject  to  a  greater 
degree  of  immediately  observable  fluctuations  in  classroom  performance. 

Since  the  FO  score  reveals  an  average  intercorrelation  with  itself 
which  is  roughly  intermediate  for  the  range  set  by  the  secondary  forms 
(L.^,  OC,  and  115) ,  we  may  infer  that  it  is  sensitive  to  a  broader  range 
of  impressions  than  the  LQ  score,  at  one  extreme,  or  the  OC  score,  at 
the  other.  This  seems  reasonable  in  view  of  the  extrapolation  required 
to  a  distant,  and  possibly  diffuse,  criterion. 

The  differences  obtained  between  the  administrative  and  research 
sets  are  minimal  as  regards  any  gain  in  reliability  using  one  as  opposed 
to  the  other.  Among  other  things,  it  may  be  suggested  that  results 
already  obtained  from  peer  nomination  studies,  where  a  research  set  was 
involved,  may  be  taken  to  ha  "real  life"  implications .  A  caution  must 
be  introduced,  however,  lest  uremature  conclusions  are  drawn  regarding 
the  differential  validity  of  forms  administered  under  these  two  sets. 
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APPENDIX  A— FORM  FO-RO 


U.  S.  NAVAL  SCHOOL,  OFFICER  CANDIDATE 
NEWPORT,  RHODE  ISLAND 


You  have  been  together  with  the  men  in  this 
From  this  contact,  you  will  have  formed  certain 
their  future  success  as  Naval  Officers. 


room  since  you  entered  OC3. 
impressions  of  them  regarding 


Considering  these  impressions,  and  carefully  weighing  the  qualities 
required  in  a  successful  Naval  Officer,  you  are  to  select  the  five  members  of 
your  section  whom  you  consider  to  have  the  highest  promise  as  Naval  Officers 
and  the  fivg  members  of  your  section  whom  you  consider  to  have  the  lowest 
promise  as  Naval  Officers.  — — — 


4Nn  OF  THESE  RATINGS  ARE  TO  BE  USED  FOR  RESEARCH  PURPOSES  ONLY 

AND  WILL  NOT  AFFECT  YOU?.  NaVY  CAREER. 

Specifically,  you  are  to  perform  these  exact  operations: 

1.  Consult  the  section  roster  which  has  been  provided  you  and 
draw  a  line  through  your  own  name. 

2.  Study  the  remaining  names  on  the  roster  and  select  the 
individual  with  the  highest  promise  of  success  as  a  Naval  Officer. 
Enter  this  name  in  the  space  labeled  H-l  below  and  then  draw  a  line 
through  that  name  on  the  roster. 


3.  Study  the  roster  again  and  then  select  the  individual  with  the 
^■weat  ££omA3e»  Enter  this  name  in  the  space  labeled  L-l  below  and 
then  draw  a  line  through  that  name  on  the  roster. 

Continue  the  study  of  this  roster,  alternately  selecting 
individuals  with  the  highest  promise  and  the  lowest  promise,  until 
you  have  entered  ten  names.  Draw  a  line  through  each  name  on  the 
roster  as  you  write  it  in  the  proper  place  here. 


HIGHEST  PROMISE _ H-1 

_ _ _ _ H-2 

- H-3 

- - - H-4 

- H-5 


- — _ L-5 

— _ L -h 

- - - L-3 

_ _  L-2 

LOWEST  PROMISE  r  n 


